For software development, regardless of the team size, the majority of research and development (R&D) time and costs are spent in testing and problems solving phases. Even if there is multitude of solutions available for automating test execution for majority of software platforms; there is no solution which automates and speeds up collection and analysis phases for the data, which can be generated during the test execution.
For example, in testing tools depending on space and a platform (e.g. Android, Windows, iOS) of interest, there are plenty of test automation and/or analysis/profiling solutions available, either commercial or open source based. Typically, these solutions are dedicated to single, small purpose, example, due to technical limitations, most of the test automation solutions are for testing a single application or code component; not the entire system. Furthermore, these test automation solutions provide a technology to automate only the test execution part, usually, the only data received from the tests is pass/fail (result). In some cases, in case of a failure, a stack trace with a limited stack depth. On the other hand, the existing analysis/profiling solutions are usually dedicated to provide help for a specific area, such as memory leak analyzer. Furthermore, analysis/profiling solutions are totally decoupled from test automation solutions, i.e. they require a user to manually record the data for analysis. Still further, they are not designed to handle large amount of data, for example, Android device can easily generate a gigabyte of data per hour, and a long test round might last hundreds of hours.
Further, existing test automation solutions are not designed for long lasting test executions. For example stability testing might require hundreds of hours of testing for software based system (or device), which is close to its sales. In such instance, most of the existing test automation solutions are not reliable and/or flexible enough for this kind of requirements. Moreover, especially with mobile/embedded devices, test automation tool/runner is quite often running inside the device (or system) under test itself. With this approach the overall test run might not be able to recover and continue testing after the system under test hangs or crashes. Another significant limitations with this approach is the data collection, i.e. if possible at all, data can be collected only to internal storage of the device, or to removable data storage (such as memory card); and fetched out from the device afterwards.
The combinations of existing solutions can be used, for example, run data collector scripts parallel with a test automation tool to get the needed data, and once all the data is available, run corresponding parsers and visualizers for each data type. When combining the complexity of all this to the fact that none of this is designed for long (can be hundreds of hours) test executions and for the huge amount of data is produces, it can be said to be impossible or at least extremely difficult to achieve the needed result (or efficiency). Some test automation tools provide primitive data collection mechanisms for few data types, but usually the technique is a built-in script which uses another tool, and thus the result, when considering efficiency, is the same as described above.
Furthermore, by generating data like described above, the data does not get synchronized in time. Some raw data formats has timestamps for data points, some does not; for those who has, the time stamp source might vary. For example, when testing an Android device, the seed for time stamps might be personal computer (PC associated with a test engineer/operator), kernel boot time, or android boot time; time precision might be in nanoseconds or in minutes. Unsynchronized data requires the person analyzing the data first of all, in the worst case, to open each type of data in a separate visualizer application, but also to manually synchronize the data between each other. For example, if a system engineer detects weird behavior in test execution video, and needs to know the lines in device (or system under test) log exactly at point of time. The video duration is relative to time in PC, on which it camera is connected to, and time stamps in device log are relative in clock in the device. Further, even if clock in PC and the device could be synchronized, the synchronization accuracy would be insufficient in most of the cases. Therefore, without interactivity between different data views, the drop in problems solving efficiency is considerable.
Generally, a common example of data logistics problem: In a test lab, one test environment finds a functional problem from a system under test. The test engineer files a bug report in which a detailed description of steps how the problem might be reproduced is provided. After the bug is filled in, at some point a system engineer is allocated to start debugging the problem and he/she notices that it requires several hours to get the system under test in the condition where it fails, and it would need several kind of debugging data from the system to pinpoint the root cause of the failure. The system engineer requests the test engineer/operator to reproduce the problem, and to collect the debugging data as well. Thereafter, the test engineer may transfer the debugging data, which might be tens of gigabytes, to a common network drive. The system engineer copies the data from the network drive to his/her PC, and starts the problems solving work. Once done, the system engineer may copy the most important subset of that data to the related bug report, and routes the bug report/ticket to a developer who is responsible of the code component where the failure happens. The developer starts work by taking local copy of the data, if it still exists in the network drive, and if notices that would need more detailed data the developer may ask more data from the system engineer and/or test engineer/operator. Finally it is again the test engineer who receives a task to reproduce the problem with more detailed debug data, and copy the data to a common network drive, from where the developer and/or the system engineer can take their local copies of it. This loop continues, and might require several iterations and more developers, until the actual problem in the code is solved. It takes huge amount of time and effort, but also requires hundreds of gigabytes of data to be transferred back and forth.
Moreover, problems occurred in the field testing for mobile devices are usually hard to reproduce in test labs, and reproducing and gathering the needed debug data during field testing is either complex or impossible. There exist conventional methods for uploading limited data, for example, application crash information, to cloud service directly from mobile devices; but no complete solution for storing and uploading all the valuable data.
Based on above, it would be desirable to have valuable debugging data in real-time from the test runs executed for a system under test, i.e. data, which helps to pinpoint a root causes of the problems for the system under test. Since, having the data on real-time enables problems solving work to start immediately, without a need to wait for test execution to finish, which may take even weeks.
Further, it would be desirable to get real-time access to all valuable debugging data produced by the system under test with minimal amount of test repetition, i.e. without a need to manually request, access and transfer the data from a test environment (system, e.g. PC, running the tests) without human interaction. The repetition of a test scenario can easily double the time spend for testing. Furthermore, without a real-time data broadcasting channel, time, effort and storage spent for transferring the needed data from person to person can be huge.
Additionally, it would be desirable to get all the data synchronized and unified: in order to quickly navigate and synchronize different data streams in time, within single viewer/analyzer application. Since, without unified and synchronized data formats, person analyzing the data needs to manually synchronize the data streams, and in worst case use different data viewers for each stream. This may take lot of time and effort, and still may be error prone and inaccurate.
Furthermore, it would be desirable to get the test system to automatically react and alarm on anomalies it detects on a system under test. The reactivity in test framework enables to dynamically start collecting more detailed data from the system under test, which decreases the need for repeating the test scenario with manually enabled data collectors. The reactive notifications enable engineers to automatically start the problem solving work, without a need to actively monitor the test run and data under interest.
Still further, it would be desirable to share any findings from the data with anyone having the access to the same data. It may save lot of time and effort if interactive links/bookmarks to exact location in the data could be shared; without transferring the complete data itself, and without a need to manually navigate to the correct location(s) in data(s) once receiving the data.
Moreover, it would be desirable to have a data viewer application, whose performance and response times remain in good level regardless of sizes of the data streams. As, the data streams even with the simplest, textual, format might get big, for example, even hundreds of gigabytes. Further, for many data stream types, there is no visualizer application at all, or exiting applications can't handle streams of big size, i.e. they either run out of memory or lose the responsiveness.
As mentioned above, test automation execution is traditionally coded in sequence of steps and each step can either pass or fail, for example, cause current test case execution to fail. Further, various systems which are automatically tested consist of many occasions that are not related to test case under execution and therefore can interrupt the test case flow. These situations are traditionally handled with if-else logic. However, as the tested systems have increasingly unexpected situations that can occur, this traditional way becomes harder to implement and maintain. Also there might be interruptions from other systems or for example notifications from different applications on the tested system that could prevent execution of the pre-written script or code.
For example, traditionally test code or test scripts have been written with if-else logic to overcome situations that may happen during the execution. If-else logic is not an effective way to handle situations that might happen nondeterministic during test case execution. Further, reactive programming approach has not been used in other test automation systems. Some tools allow launching of certain scripts for cleaning up the tested systems state after a failure has happened in the test case execution, but this just an implementation of if-else logic.
Furthermore, state analysis in object recognition based test automation tools has traditionally been using the XML representation of the user interface (UI) on the device screen. An XPath is well known method to analyze the content of the XML document. There is no known implementation where XML and XPath or similar (JSON etc.) approach would have been used in checking the conditions when the UI state changes. The XPath is not ideal way since it is constructed with a query string and complex queries tend to be cumbersome to update and maintain.
In addition to above, test specification describes how a functionality may be tested. The test specification defines the input and expected output in the system under test. In test automation, this specification is scripted or programmed with scripting library calls specific to test automation framework, and the result of the test execution is determined within the scripting library and test framework. When there's different kind of systems under test and same test specifications is used to test each system, test automation scripts might not work since the different systems require different kind of inputs to reach expected output. Thus, the test scripts need to have multiple versions for each system under test. This creates maintenance need, i.e. when a test script needs to be changed in some way the change needs to be replicated in each version of the script available.
A solution to aforementioned problem may be keyword or action based test automation to quickly adapt to changes in the tested system. For example, if the same keyword or action is used in various test scripts, then the changes in the keyword or action will be replicated to all test scripts. However, there is no method existing in the field of test automation that can override the defined scripting library methods (e.g. keywords) by automatically selecting a location of the overrides.
Additionally, the system or (device under test) may freeze (or its UI may freeze) in a situation when a user controlled system stops responding to user interactions with UI. For example, if the user controlled system includes a display, the display or touch/user input module might stop functioning while the software layer is still working. The UI freezes are the most fatal errors from end user point of view since the only way to recover from it is to reboot the device. Further, such type of errors are often hard to debug and repeat in the testing process. For example, when software is tested by recognizing objects from the software level, this type of errors are hard to detect since the test verification relies on the information received from the system under testing and tests can be conducted without physically interacting with the device. Therefore, UI test automation methods that work with bitmap captured from system under test (via system software) or software object recognition do not have a method to detect the described freeze situations automatically.
Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks of conventional testing methods and systems for software based systems.